1. Field of the Invention
The present invention generally relates to computer systems, and more particularly to a method and apparatus for memory access by an input/output (I/O) device of a computer system which uses translation control entries (TCEs) in system memory and copies of the TCEs stored in a local cache of the I/O device.
2. Description of the Related Art
A typical structure for a conventional computer system includes one or more processing units connected to a system memory device (random access memory or RAM) and to various peripheral, or input/output (I/O) devices such as a display monitor, a keyboard, a graphical pointer (mouse), and a permanent storage device (hard disk). The system memory device is used by a processing unit in carrying out program instructions, and stores those instructions as well as data values that are fed to or generated by the programs. A processing unit communicates with the other components by various means, including one or more interconnects (buses), or direct memory-access channels. A computer system may have many additional components, such as serial, parallel or universal serial bus (USB) ports for connection to, e.g., printers, and network adapters. Other components might further be used in conjunction with the foregoing; for example, a display adapter might be used to control a video display monitor, a memory controller can be used to access the system memory, etc.
Several different bus designs have been developed for interconnecting the various computer components. The original personal computer (PC) introduced by International Business Machines Corp. used an “expansion” bus referred to as the XT bus, which allowed a user to add various optional devices, such as additional memory (RAM), sound cards, telephone modems, etc. This early design was improved upon by adding more data and address lines, new interrupt lines, and direct memory-access (DMA) control lines, to create the AT bus, which is also referred to as the Industry Standard Architecture (ISA) bus. The AT design allowed the microprocessor to run at a faster speed than the expansion bus. A 32-bit extension to this bus was later created, which is referred to as the Extended Industry Standard Architecture (EISA).
In addition to the foregoing designs, several other bus designs have been developed allowing the use of a system bus which interconnects the processors and system memory devices, along with a separate bus which interconnects the peripheral devices to the system bus (using a bus bridge). Two well-known standards are the Video Electronics Standards Association (VL) bus, and the Peripheral Component Interconnect (PCI) bus. The PCI specification allows multiple PCI-compliant expansion cards to be installed in slots constructed along the PCI bus. A PCI local bus system uses a PCI controller. A PCI controller can exchange data with the microprocessor 64 bits at a time, and allows “intelligent” PCI-compliant adapters to perform tasks concurrently with the microprocessor, using a technique called bus mastering. The PCI specification also allows for multiplexing, a technique that permits more than one electrical signal to be present on the bus at one time.
An extension to the PCI specification referred to as PCI Express (or PCIe) has been created which provides PCI compatibility using established PCI programming models and further allows new hardware to take advantage of enhanced features. The PCIe architecture provides a low-pin count interface and support for multiple interconnect widths, new hot-plug cards, and higher transfer rates. There are also supplemental specifications to the PCIe architecture for delivery of increased performance capabilities which consider issues such as power consumption and heat management.
One enhancement made available with PCIe systems is address translation services (ATS). The computer system uses a translation agent to translate addresses from a virtual address provided by the I/O device to a physical address in system memory. A PCIe endpoint (I/O device) may cache an address translation entry from the system-level address translation and protection table (ATPT) that resides in memory. An ATPT entry is also referred to as a translation control entry (TCE). Caching of a TCE allows an endpoint to request a DMA transfer using an address that it pre-translates, so the TCE does not have to be fetched from memory in series with the DMA operation, and hence reduces the latency of the operation.
When the memory system deems that the translation in a system TCE is no longer desired or needs to be changed, the TCE is invalidated. At that point, the cached entry in the PCIe address translation cache (ATC) must be invalidated as well. The ATS specification provides protocols on the PCIe links to carry out this invalidation; however, there is a latency in carrying out the invalidation of the TCE, i.e., from the time the system decides to invalidate the entry until the entry is actually invalidated, and the ATS specification does not provide any guidance on how the system is supposed to handle this latency. Software and hardware in the PCIe system require certain latencies to be associated with the TCE invalidation process, but ATS introduces new and indeterminate invalidation latencies. It can accordingly be difficult to ascertain when the system can reuse a TCE that is being invalidated from another process. It is also important to be able to prevent further caching of the TCE by an endpoint while the invalidation process is taking place. It would, therefore, be desirable to devise an improved method for TCE invalidation that could synchronize the system level and the I/O device. Since the delay between system memory and the I/O device can be lengthy, it would be further advantageous if the method could accommodate the asynchronous nature of the invalidation operation.